In [1]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import warnings
warnings.filterwarnings('ignore')
In [2]:
#reading the dataset
vehicle_df = pd.read_csv('vehicle.csv')
In [3]:
#First few entries of dataset
vehicle_df.head(10)
Out[3]:
In [4]:
vehicle_df.info()
- There are missing values as NaN values in the dataset
Attribute Information:
ATTRIBUTES
COMPACTNESS (average perim)**2/area
CIRCULARITY (average radius)**2/area
DISTANCE CIRCULARITY area/(av.distance from border)**2
RADIUS RATIO (max.rad-min.rad)/av.radius
PR.AXIS ASPECT RATIO (minor axis)/(major axis)
MAX.LENGTH ASPECT RATIO (length perp. max length)/(max length)
SCATTER RATIO (inertia about minor axis)/(inertia about major axis)
ELONGATEDNESS area/(shrink width)**2
PR.AXIS RECTANGULARITY area/(pr.axis length*pr.axis width)
MAX.LENGTH RECTANGULARITY area/(max.length*length perp. to this)
SCALED VARIANCE (2nd order moment about minor axis)/area ALONG MAJOR AXIS
SCALED VARIANCE (2nd order moment about major axis)/area ALONG MINOR AXIS
SCALED RADIUS OF GYRATION (mavar+mivar)/area
SKEWNESS ABOUT (3rd order moment about major axis)/sigma_min**3 MAJOR AXIS
SKEWNESS ABOUT (3rd order moment about minor axis)/sigma_maj**3 MINOR AXIS
HOLLOWS RATIO (area of hollows)/(area of bounding polygon)
Where sigma_maj2 is the variance along the major axis and sigma_min2 is the variance along the minor axis, and
area of hollows= area of bounding poly-area of object
The area of the bounding polygon is found as a side result of the computation to find the maximum length. Each individual length computation yields a pair of calipers to the object orientated at every 5 degrees. The object is propagated into an image containing the union of these calipers to obtain an image of the bounding polygon.
NUMBER OF CLASSES 3
VAN, BUS, CAR
- The target variable is an object with different models of vehicles
In [5]:
vehicle_df.describe(include = 'all').transpose()
Out[5]:
- From the data it seems that there are some missing values for some of the features
In [6]:
vehicle_df.isnull().count()
Out[6]:
- There are no null values in the dataset
In [7]:
#Copying the dataset into another dataframe
vehicle_df1 = vehicle_df.copy(deep = True)
In [8]:
vehicle_df1.head()
Out[8]:
- There are very few NaN values in the dataset. Let us replace them with 'zeroes'
- MISSING VALUES
In [9]:
#Replacing the NaN values with the median values as mean is sensitive to outliers
columns_missingvalue = ['compactness','circularity','distance_circularity','radius_ratio','pr.axis_aspect_ratio',
'max.length_aspect_ratio','scatter_ratio','elongatedness','pr.axis_rectangularity',
'max.length_rectangularity','scaled_variance','scaled_variance.1','scaled_radius_of_gyration',
'scaled_radius_of_gyration.1','skewness_about','skewness_about.1','skewness_about.2','hollows_ratio'
]
for i in columns_missingvalue:
median = vehicle_df1[i].median()
vehicle_df1[i].replace(np.nan,median,inplace = True)
In [10]:
vehicle_df1.info()
- All missing values are replaced by median values
Label encoding of Target class¶
In [11]:
# Label encoding of the target class is done as most of the models require the attributes and target variables to be numbers
vehicle_df1['class'] = pd.Categorical(vehicle_df1['class']).codes
In [12]:
vehicle_df1.info()
- After converting the 'class' column to codes, it is converted to int datatype from object which is desirable in modelling also
In [13]:
vehicle_df1.head(10)
Out[13]:
In [14]:
vehicle_df1.plot(kind = 'box',figsize = (30,30),subplots = True, layout = (5,4))
Out[14]:
- There are outliers at both the top and the bottom range of box plots in some of the attributes which are to be addressed. These outliers if not attended will influence the models used for prediction. Hence, letus replace these outliers with the median values which willnot effect the distribution also.
DEALING WITH OUTLIERS¶
In [15]:
columns_outlier = ['compactness','circularity','distance_circularity','radius_ratio','pr.axis_aspect_ratio',
'max.length_aspect_ratio','scatter_ratio','elongatedness','pr.axis_rectangularity',
'max.length_rectangularity','scaled_variance','scaled_variance.1','scaled_radius_of_gyration',
'scaled_radius_of_gyration.1','skewness_about','skewness_about.1','skewness_about.2','hollows_ratio'
]
for i in columns_outlier:
dp75, dp25 = np.percentile(vehicle_df1[i], [75 ,25])
IQR = dp75 - dp25
Up_range = dp75 + 1.5*IQR
Low_range = dp25 - 1.5*IQR
median = vehicle_df1[i].median()
print('Number of upper range outliers is : ' ,vehicle_df1[vehicle_df1[i] > Up_range][i].shape[0])
print('Number of Lower range outliers is : ' , vehicle_df1[vehicle_df1[i] < Low_range][i].shape[0])
vehicle_df1.loc[(vehicle_df1[i] > Up_range),i] = median
vehicle_df1.loc[(vehicle_df1[i] < Low_range),i] = median
print('All outliers are replaced of ', i)
print('*'*100)
- All the outliers are replaced with median values as it doesnot effect the distribution of these variables
In [16]:
# Viewing the dataframe after eliminating outliers by replacing with median values
vehicle_df1.head(10)
Out[16]:
In [17]:
# Checking boxplots for removal of outliers
vehicle_df1.plot(kind = 'box',figsize = (30,30),subplots = True, layout = (5,4))
Out[17]:
- All outliers are replaced and there are no outliers as depicted by above boxplots
- NORMALIZATION OF DATA
In [18]:
#For processing of data
from sklearn import preprocessing
In [19]:
x_array = np.array(vehicle_df1['compactness'])
norm_compactness = preprocessing.normalize([x_array])
x_array1 = np.array(vehicle_df1['circularity'])
norm_circularity = preprocessing.normalize([x_array1])
x_array2 = np.array(vehicle_df1['distance_circularity'])
norm_distance_circularity = preprocessing.normalize([x_array2])
x_array3 = np.array(vehicle_df1['radius_ratio'])
norm_radius_ratio = preprocessing.normalize([x_array3])
x_array4 = np.array(vehicle_df1['pr.axis_aspect_ratio'])
norm_praxis_aspect_ratio = preprocessing.normalize([x_array4])
x_array5 = np.array(vehicle_df1['max.length_aspect_ratio'])
norm_maxlength_aspect_ratio = preprocessing.normalize([x_array5])
x_array6 = np.array(vehicle_df1['scatter_ratio'])
norm_scatter_ratio = preprocessing.normalize([x_array6])
x_array7 = np.array(vehicle_df1['elongatedness'])
norm_elongatedness = preprocessing.normalize([x_array7])
x_array8 = np.array(vehicle_df1['pr.axis_rectangularity'])
norm_praxis_rectangularity = preprocessing.normalize([x_array8])
x_array9 = np.array(vehicle_df1['max.length_rectangularity'])
norm_maxlength_rectangularity = preprocessing.normalize([x_array9])
x_array10 = np.array(vehicle_df1['scaled_variance'])
norm_scaled_variance = preprocessing.normalize([x_array10])
x_array11 = np.array(vehicle_df1['scaled_variance.1'])
norm_scaled_variance1 = preprocessing.normalize([x_array11])
x_array12 = np.array(vehicle_df1['scaled_radius_of_gyration'])
norm_scaled_radius_of_gyration = preprocessing.normalize([x_array12])
x_array13 = np.array(vehicle_df1['scaled_radius_of_gyration.1'])
norm_scaled_radius_of_gyration1 = preprocessing.normalize([x_array13])
x_array14 = np.array(vehicle_df1['skewness_about'])
norm_skewness_about = preprocessing.normalize([x_array14])
x_array15 = np.array(vehicle_df1['skewness_about.1'])
norm_skewness_about1 = preprocessing.normalize([x_array15])
x_array16 = np.array(vehicle_df1['skewness_about.2'])
norm_skewness_about2 = preprocessing.normalize([x_array16])
x_array17 = np.array(vehicle_df1['hollows_ratio'])
norm_hollows_ratio = preprocessing.normalize([x_array17])
x_array18 = np.array(vehicle_df1['class'])
In [20]:
vehicle_df_norm = pd.DataFrame({'compactness': norm_compactness[0,:], 'circularity': norm_circularity[0,:], 'distance_circularity': norm_distance_circularity[0,:], 'radius_ratio': norm_radius_ratio[0,:],
'pr.axis_aspect_ratio' : norm_praxis_aspect_ratio[0,:], 'max.length_aspect_ratio' : norm_maxlength_aspect_ratio[0,:], 'scatter_ratio' : norm_scatter_ratio[0,:],
'elongatedness' : norm_elongatedness[0,:], 'pr.axis_rectangularity' : norm_praxis_rectangularity[0,:], 'max.length_rectangularity' : norm_maxlength_rectangularity[0,:],
'scaled_variance' : norm_scaled_variance[0,:], 'scaled_variance.1' : norm_scaled_variance1[0,:], 'scaled_radius_of_gyration' : norm_scaled_radius_of_gyration[0,:],
'scaled_radius_of_gyration.1' : norm_scaled_radius_of_gyration1[0,:], 'skewness_about' : norm_skewness_about[0,:],'skewness_about.1' : norm_skewness_about1[0,:],
'skewness_about.2' : norm_skewness_about2[0,:], 'hollows_ratio' : norm_hollows_ratio[0,:], 'class' : x_array18
})
vehicle_df_norm.head()
Out[20]:
In [21]:
vehicle_df_norm.hist(figsize = (20,20))
Out[21]:
In [22]:
plt.figure(figsize = (20,20))
sns.pairplot(vehicle_df_norm,diag_kind = 'kde',hue = 'class')
Out[22]:
- The pair plot with density plots in the diagonal shows that there are density plots with almost no tails, as we have removed outliers
- The pair panel reveals that there are many plots which shows good linear relationship which is essential for the PCA to be implemented successfully
In [23]:
corr = vehicle_df_norm.corr()
corr
Out[23]:
- The PCA analysis will be helpfull modelling as many of the correlations are more than 0.5
In [24]:
plt.figure(figsize = (20,20))
sns.heatmap(corr,annot = True,linewidth = 1)
Out[24]:
- There are many such cells in the heatmap which have correlation greater than 0.5. PCA will be of good help in reducing the dimensions
In [25]:
print(vehicle_df_norm['class'].value_counts())
In [26]:
sns.countplot(x = 'class', data = vehicle_df_norm)
Out[26]:
- Boxplot of the Target variable 'Class' against the other attributes
In [27]:
col = vehicle_df_norm[:17]
for item in col:
plt.title('Class' + ' against ' + str(item))
sns.boxplot(x = vehicle_df_norm['class'], y = vehicle_df_norm[item], data = vehicle_df_norm)
plt.show()
Splitting the data set¶
In [83]:
#To split the data into train and test data set
from sklearn.model_selection import train_test_split
from sklearn.model_selection import cross_val_score, cross_val_predict
from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import KFold
#To check the accuracy of the model
from sklearn.metrics import accuracy_score
# Making the Confusion Matrix
from sklearn.metrics import confusion_matrix
# for prepartation of classification report
from sklearn.metrics import classification_report,make_scorer
# for drawing roc and aoc curves
from sklearn.metrics import confusion_matrix, roc_curve, roc_auc_score,auc
# for ensemble techniques
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier, AdaBoostClassifier
In [84]:
X = vehicle_df_norm.iloc[:,:18]
y = vehicle_df_norm.iloc[:,18]
In [85]:
X.head(10)
Out[85]:
In [86]:
y.head(10)
Out[86]:
In [87]:
#y = label_binarize(y, classes=[0, 1, 2])
#n_classes = y.shape[1]
- Standardizing the feature variables using StandardScaler
In [88]:
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from scipy.stats import zscore
In [89]:
sc = StandardScaler()
X_std = sc.fit_transform(X)
PCA ANALYSIS¶
- Creating Covariance Matrix
In [90]:
# Obtaining the co-variance matrix
cov_matrix = np.cov(X_std.T)
In [91]:
print('The Covariance matrix is \n%s', cov_matrix)
In [92]:
# Step 2 : Calculating eigenvalues and eigenvectors
eigenvalues,eigenvectors = np.linalg.eig(cov_matrix)
In [93]:
print('Eigen values are \n%s', eigenvalues)
In [94]:
print('Eigen Vectors are \n%s', eigenvectors)
In [95]:
# Step 3 : Sort eigenvalues in descending order
# Make a set of (eigenvalue, eigenvector) pairs
eig_pairs = [(eigenvalues[index], eigenvectors[:,index]) for index in range(len(eigenvalues))]
# Sort the (eigenvalue, eigenvector) pairs from highest to lowest with respect to eigenvalue
eig_pairs.sort()
eig_pairs.reverse()
print(eig_pairs)
# Extract the descending ordered eigenvalues and eigenvectors
eigvalues_sorted = [eig_pairs[index][0] for index in range(len(eigenvalues))]
eigvectors_sorted = [eig_pairs[index][1] for index in range(len(eigenvalues))]
# Let's confirm our sorting worked, print out eigenvalues
print('Eigenvalues in descending order: \n%s' %eigvalues_sorted)
In [96]:
tot = sum(eigenvalues)
var_explained = [(i / tot) for i in sorted(eigenvalues, reverse=True)] # an array of variance explained by each
# eigen vector... there will be 9 entries as there are 9 eigen vectors)
cum_var_exp = np.cumsum(var_explained) # an array of cumulative variance. There will be 9 entries with 9 th entry
# cumulative reaching almost 100%
cum_var_exp
Out[96]:
In [97]:
plt.bar(range(1,19), var_explained, alpha=0.5, align='center', label='individual explained variance')
plt.step(range(1,19),cum_var_exp, where= 'mid', label='cumulative explained variance')
plt.ylabel('Explained variance ratio')
plt.xlabel('Principal components')
plt.legend(loc = 'best')
plt.show()
- First 10 principal components explain 98% of the variance and after that the covariance curve is almost a straight line. Hence, let us consider only the first 10 principal components
Reducing the number of principal components for modelling¶
In [98]:
P_reduced = np.array(eigvectors_sorted[0:10])
In [99]:
P_reduced
Out[99]:
In [100]:
X_std_10D = np.dot(X_std,P_reduced.T)
X_std_10D
Out[100]:
In [169]:
#The feature matrix with pricipal components
PCA_X = pd.DataFrame(X_std_10D)
PCA_X.head(10)
Out[169]:
In [102]:
PCA_X.corr()
Out[102]:
Splitting the dataset into training and test data using PCA vectors¶
In [103]:
seed = 1000
X_train, X_test, y_train, y_test = train_test_split(PCA_X,y,test_size = 0.3,random_state = seed)
Training and Kfold validation of classical algorithms¶
In [105]:
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC
from sklearn.preprocessing import label_binarize
from scipy import interp
from itertools import cycle
In [106]:
# Defining Confusion Matrix
def confusion(x,y):
cm = confusion_matrix(x,y)
f, ax = plt.subplots(figsize=(5, 5))
sns.heatmap(cm, annot=True, fmt="d", linewidths=.5, ax=ax)
plt.title("Confusion Matrix", fontsize=20)
plt.subplots_adjust(left=0.1, right=0.5, bottom=0.11, top=0.5)
ax.set_yticks(np.arange(cm.shape[0]) + 0.5, minor=False)
ax.set_xticklabels("")
#ax.set_yticklabels(['Refused T. Deposits', 'Accepted T. Deposits'], fontsize=10, rotation=360)
plt.show()
In [107]:
# Initiating Models
clfLR = LogisticRegression(random_state = seed, class_weight = None)
clfNB = GaussianNB()
clfKNN = KNeighborsClassifier()
clfDT = DecisionTreeClassifier(random_state = seed, class_weight = None)
clfSVC = SVC(random_state = seed, class_weight = None)
clfRF = RandomForestClassifier(n_estimators = 18)
clfGB = GradientBoostingClassifier()
In [108]:
# Creating the matrix of models
Models = []
Models.append(('Logistic Regression', clfLR))
Models.append(('Naive Bayes', clfNB))
Models.append(('K-Nearest Neighbour',clfKNN))
Models.append(('Decision Tree', clfDT))
Models.append(('Support Vector Classifier', clfSVC))
Models.append(('Random Forest',clfRF))
Models.append(('Gradeint Boosting', clfGB))
In [112]:
#Training and Cross validation of different models with confusion matrix, classification report and AUC of ROC-AUC curve
results = []
names = []
mean_score = []
fpr = dict()
#initialising the kfold value for Logistic Regression
for name,model in Models:
kfold = KFold(n_splits=5, shuffle=True, random_state = seed)
cv_results = cross_val_score(model,X_train, y_train,cv = kfold, scoring = 'accuracy')
cv_predict_train = cross_val_predict(model,X_train,y_train,cv = kfold)
confusion(y_train,cv_predict_train)
cr = classification_report(y_train, cv_predict_train)
print(cr)
results.append(cv_results)
names.append(name)
mean_score.append(cv_results.mean())
msg = "%s:Mean of cross validation accuracy %f" % (name,cv_results.mean())
print(msg)
In [113]:
#Defining dataframe of classifiers with respective cross validation score mean and AUC of ROC-AUC curve
dtrain = {'Classifiers': ['Logistic Regession', 'Naive Bayes', 'K Nearest Neighbor', 'Decision Tree', 'Support Vector Classifier',
'Random Forest', 'Gradient Boosting'],
'Cross Val Mean Score' : np.array(mean_score),
# 'Area Under ROC Curve' : np.array(AUC)
}
Classifier_train = pd.DataFrame(data = dtrain)
Classifier_train
Out[113]:
In [114]:
Models
Out[114]:
In [115]:
# Displaying models in the ascending order of their mean scores in cross validation
Classifier_train = Classifier_train.sort_values(by=['Cross Val Mean Score'], ascending=False)
Classifier_train
Out[115]:
- The support vector classifier has the highest mean accuracy, followed by Gradient Boosting, KNN
Testing the Algorithms¶
In [116]:
# Testing the models
Models = []
Models.append(('Logistic Regression', clfLR))
Models.append(('Naive Bayes', clfNB))
Models.append(('K-Nearest Neighbour',clfKNN))
Models.append(('Decision Tree', clfDT))
Models.append(('Support Vector Classifier', clfSVC))
Models.append(('Random Forest',clfRF))
Models.append(('Gradeint Boosting', clfGB))
In [117]:
#Fitting the models trained, testing, their confusion matrix, classification report
names = []
score_test = []
for name,model in Models :
model.fit(X_test,y_test)
y_predict_test = model.predict(X_test)
confusion(y_test,y_predict_test)
test_score = model.score(X_test,y_test)
score_test.append(test_score)
cr = classification_report(y_test,y_predict_test)
print(cr)
msg = "%s : Test Score is %.2f" %(name,test_score)
print(msg)
print('*' *100)
In [118]:
#Defining data frame of Models and their test scores
dtest = {'Classifiers': ['Logistic Regession', 'Naive Bayes', 'K Nearest Neighbor', 'Decision Tree', 'Support Vector Classifier',
'Random Forest', 'Gradient Boosting'],
'Test Score' : np.array(score_test)
}
Classifier_test = pd.DataFrame(data = dtest)
In [119]:
#Defining dataset for test score and displaying in the order of test score
Classifier_test = Classifier_test.sort_values(by=['Test Score'], ascending=False)
Classifier_test
Out[119]:
- Decision Tree, Random Forest and Gradient Boosting are overfitting in the training set.
GRID SEARCH¶
In [120]:
#import gridsearch from sklearn
from sklearn.model_selection import GridSearchCV
LOGISTIC REGRESSION
In [121]:
#Initial Logistic Tree Classifier
clfLR
Out[121]:
- Initial hyper parameter value used in grid search in default value of C i.e., 1.0
In [122]:
param_grid = {
'C' : [0.01,0.1,1,10,100]
}
In [123]:
grid_search_LR = GridSearchCV(estimator = clfLR, param_grid = param_grid, scoring = None, cv = 3, n_jobs = -1, verbose = 3)
In [124]:
grid_search_LR.fit(X_train,y_train)
Out[124]:
In [125]:
grid_search_LR.best_estimator_
Out[125]:
In [126]:
grid_search_LR.best_score_
Out[126]:
In [127]:
y_predict_gs_LR = grid_search_LR.predict(X_test)
In [128]:
confusion(y_test,y_predict_gs_LR)
In [129]:
print(classification_report(y_test,y_predict_gs_LR))
DECISION TREE CLASSIFIER
In [130]:
# Initial Decision Tree Classifier
clfDT
Out[130]:
In [131]:
param_grid = {
'criterion': ['gini', 'entropy'],
'min_samples_split': [2,4,10,20],
'min_samples_leaf': [1,5,10,20],
'max_depth': [2,5,10,20]
}
In [132]:
grid_search = GridSearchCV(estimator = clfDT, param_grid = param_grid, scoring = 'accuracy', cv = 3, n_jobs = -1, verbose = 3)
In [133]:
grid_search.fit(X_train,y_train)
Out[133]:
In [134]:
grid_search.best_estimator_
Out[134]:
In [135]:
grid_search.best_score_
Out[135]:
In [136]:
y_predict_gs = grid_search.predict(X_test)
In [137]:
confusion(y_test,y_predict_gs)
- The overfitting of decision tree is being corrected after grid search, and the accuracy comes down to 83%
RANDOM FOREST
In [138]:
# Initial Random forest classifier
clfRF
Out[138]:
In [139]:
param_grid = { "criterion" : ["gini", "entropy"],
"min_samples_leaf" : [1, 5, 10],
"min_samples_split" : [2, 4, 10,],
"n_estimators": [50, 100, 400]
}
In [140]:
gs_rf = GridSearchCV(estimator=clfRF, param_grid=param_grid, scoring='accuracy', cv=3, n_jobs=-1, verbose = 1)
In [141]:
gs_rf=gs_rf.fit(X_train, y_train)
In [142]:
gs_rf.best_estimator_
Out[142]:
In [143]:
gs_rf.best_score_
Out[143]:
In [144]:
y_predict_gs_rf = gs_rf.predict(X_test)
In [145]:
confusion(y_test,y_predict_gs_rf)
In [146]:
print(classification_report(y_test,y_predict_gs_rf))
- The overfitting of the random forest algorithm is addressed and the accuracy comes down to 91% after grid search
SUPPORT VECTOR CLASSIFIER
In [147]:
#INITIAL SVC CLASSIFIER
clfSVC
Out[147]:
In [148]:
param_grid = {'C': [0.1,1, 10, 100],
'gamma': [1,0.1,0.01,0.001],
'kernel': ['rbf','sigmoid']}
In [149]:
gs_svc = GridSearchCV(estimator=clfSVC, param_grid=param_grid, scoring='accuracy', cv=3, n_jobs=-1,verbose=3)
In [150]:
gs_svc.fit(X_train,y_train)
Out[150]:
In [151]:
gs_svc.best_estimator_
Out[151]:
In [152]:
gs_svc.best_score_
Out[152]:
In [153]:
y_predict_gs_svc = gs_svc.predict(X_test)
In [154]:
confusion(y_test,y_predict_gs_svc)
In [155]:
print(classification_report(y_test,y_predict_gs_svc))
- Error due to variance is reduced after grid search and accuracy comes down to 94% from 97%
BAGGING CLASSIFIER¶
In [156]:
from sklearn.ensemble import BaggingClassifier
In [157]:
Models = []
Models.append(('Logistic Regression', clfLR))
Models.append(('Naive Bayes', clfNB))
Models.append(('K-Nearest Neighbour',clfKNN))
Models.append(('Decision Tree', clfDT))
Models.append(('Support Vector Classifier', clfSVC))
Models.append(('Random Forest',clfRF))
Models.append(('Gradeint Boosting', clfGB))
In [158]:
name = []
Bagging_score = []
for name,model in Models:
model_bagging = BaggingClassifier(base_estimator = model,n_estimators=50,max_samples=.7 , oob_score=True,random_state = seed,n_jobs = -1)
y_predict_bagging = model_bagging.fit(X_train,y_train)
score_bagging = model_bagging.oob_score_
Bagging_score.append(score_bagging)
print('Bagging score of model %s is %.3f' %(name,score_bagging))
In [159]:
Bagging_score
Out[159]:
In [160]:
d_test_bagging = {'Classifiers': ['Logistic Regession', 'Naive Bayes', 'K Nearest Neighbor', 'Decision Tree', 'Support Vector Classifier',
'Random Forest', 'Gradient Boosting'],
'Test Score' : np.array(score_test),
'Bagging Score' : np.array(Bagging_score)
}
Classifier_test_bagging = pd.DataFrame(data = d_test_bagging)
In [161]:
Classifier_test_bagging
Out[161]:
In [162]:
Models
Out[162]:
- The bagging classifier improves the performance of almost all the algorithms, reducing the error due to variance of all the models
ADAPTIVE BOOSING¶
In [163]:
from sklearn.ensemble import AdaBoostClassifier
In [164]:
Models = []
Models.append(('Logistic Regression', clfLR))
Models.append(('Naive Bayes', clfNB))
Models.append(('Decision Tree', clfDT))
In [165]:
name = []
Boosting_score = []
for name,model in Models:
model_boosting = AdaBoostClassifier(base_estimator=model, n_estimators=50, random_state = seed,algorithm = 'SAMME.R')
model_boosting.fit(X_train,y_train)
score_boosting = model_bagging.oob_score_
Boosting_score.append(score_boosting)
print('Boosting score of model %s is %.3f' %(name,score_boosting))
In [166]:
d_test_boosting = {'Classifiers': ['Logistic Regession', 'Naive Bayes', 'Decision Tree'],
'Boosting Score' : np.array(Boosting_score)
}
Classifier_test_boosting = pd.DataFrame(data = d_test_boosting)
In [167]:
Classifier_test_boosting
Out[167]:
In [168]:
Models
Out[168]:
- The adaptive boosting improves the accuracy of logistic regression, Naive Bayes and Decision tree
Conclusions¶
- The missing values and the outliers in the dataset are imputed with the central values (median values) as it will not change the distribution of attributes predominantly.
- The principal component analysis (PCA) is done and 10 of the 18 principal components are taken which explains about 98.4% of the variance in the dataset.
- Logistic Regression, KNearest Neighbours, Naive Bayes, Decision tree, Random Forest, Support Vector Classfier and Gradient descent models are checked in modelling with 5 fold cross validation in training phase. Clarification report and confusin matrix of each model is obtained with the Principal components selected.
- Support vector classifier, K Nearest neigbor, Logistic regression and naive Baye's does well in both training and testing phase in terms of accuracy
- Decision tree, Random forest and Gradient Boosting algorithms found to give overfit models in the testing phase
- The Grid Search is used to select the hyper parameters of Logistic regression, Decision tree, random forest and support vector classifier.
- The overfit problem of decision tree, Random Forest and Gradient Boosting is solved by fitting on the best fit model obtained during the testing phase
- The Ensemble technique Bagging is applied on all the base models used in modelling of the dataset. It was found that the Bagging solves the overfit problem of Decision Tree, Random Forest and Gradient Boosing as it was giving a realistic scores of the model.
- Bagging was found to decrease the error due to bias on all the models, as the score on base model is more than the score using the bagging ensemble technique
- Adapting Boosting Ensemble is used on Logistic Regression, Naive Bayes and Decision tree which supports the weighted sample method requied for Adaboosting. It was found that the score of all these models increased compared to the base scores.
- Finally, Support Vector Classifier is found to be best model for the dataset as it does well in both training and the testing with almost equal scores and fairly good recall value
In [ ]:
In [ ]: